Skip to content

feat: add stale-on-error Redis recovery - #121

Draft
lan17 wants to merge 1 commit into
mainfrom
agent/stale-on-error
Draft

feat: add stale-on-error Redis recovery#121
lan17 wants to merge 1 commit into
mainfrom
agent/stale-on-error

Conversation

@lan17

@lan17 lan17 commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Summary

Add opt-in stale-on-error recovery from a physically retained Redis value, reconciled onto the current v0.20 native-I/O, compression, coalescing, shadow, and GLIDE 2.x architecture.

  • F = ttlSec[CacheLayer.REMOTE] remains the logical fresh age.
  • M = staleOnErrorMaxAgeSec is the absolute recovery age.
  • Ordinary reads serve only 0 <= age < F.
  • After a definitive ordinary miss and a source-of-truth rejection, one bounded reread may serve 0 <= age < M.
  • Recovered data is returned without refreshing Redis, populating process-local cache, or starting shadow work.

Closes #117

Contract and flow

age < F        fresh
F <= age < M   retained; eligible only after the source rejects
age >= M       unavailable
flowchart TD
  A[Redis read with maxAge F] -->|hit| B[Return fresh]
  A -->|definitive miss| C[Call source of truth]
  A -->|error or timeout| D[Call source; recovery forbidden]
  C -->|success| E[Return and publish normally with Redis PX M]
  C -->|rejection| F[Redis reread with maxAge M]
  F -->|eligible| G[Return retained value without publication]
  F -->|miss, error, timeout, or decode failure| H[Throw identical source rejection]
  D -->|rejection| H
Loading

All source rejections qualify, including arbitrary rejection values and FallbackTimeoutError. Recovery gets a fresh effective remoteReadTimeoutMs budget. Existing coalescing shares the complete read/source/recovery chain; coalesce: false preserves independent chains. A recovered value may be memoized only inside the already-active request-local scope.

Configuration

new DialCacheKeyConfig({
  ttlSec: { [CacheLayer.REMOTE]: 300 }, // F
  staleOnErrorMaxAgeSec: 3_600,         // M
});
  • Omitted: disabled by default; inherits through a sparse runtime overlay.
  • 0: explicitly disables an inherited policy.
  • Positive: enables recovery and requires 0 < F < M <= 31,536,000 seconds.
  • Static invalid combinations fail fast.
  • Invalid runtime M records config_resolution, disables only recovery for that invocation, and preserves an otherwise valid fresh Redis policy.
  • DialCacheKeyConfig.disabled() explicitly sets M to 0.

The invocation's once-resolved F/M snapshot governs both reads. Lowering a boundary takes effect immediately. Raising M cannot resurrect or extend a key written with a shorter physical TTL; only a later successful write gets the longer retention.

Redis protocol and adapters

This keeps v0.20's native payload I/O; it does not restore the pre-v0.20 read Lua scripts.

Every semantic read enqueues an ordered same-primary pair in one pipeline/batch round trip:

  1. native GET, or atomic tracked MGET(value, watermark);
  2. Redis TIME on the same connection/primary.

The adapter decodes the full DecodedRedisFrame in Node and accepts it only when 0 <= serverNowMs - createdAtMs < maxAgeMs. Future, unsafe, malformed, unsupported, and watermark-fenced frames are misses or typed protocol errors according to the existing contract. Normal, shadow, and confirmation reads always pass F, even when recovery is off; the post-source recovery reread alone passes M.

Both write modes keep payload bytes out of Lua:

  1. native SET PX writes a version-0 payload placeholder with a per-write nonce;
  2. a small stamp script verifies that nonce and promotes the header with Redis server time.

The tracked stamp also fences the invalidation watermark and maintains its TTL. A lost or raced placeholder fails honestly instead of promoting another writer's frame. Node-redis and GLIDE preserve SET-error precedence and recover stamp scripts only after NOSCRIPT.

RedisReadRequest.maxAgeMs and DialCacheRedisClient.enforcesMaxAge: true are required. The constructor rejects old/custom clients that do not attest to the logical-age contract. Shared protocol helpers and both stamp sources are exported through dialcache/redis-protocol; packed TypeScript, ESM, and CommonJS fixtures cover them.

Failure, invalidation, shadow, and publication safety

  • Recovery is attempted only after a definitive initial miss. An initial Redis error or timeout never triggers a second Redis operation.
  • An initial payload deserialization/decompression failure is not reread as stale.
  • Recovery miss/error/timeout/decode failure always rethrows the exact original source rejection.
  • The tracked reread observes the current watermark, so invalidation during the source attempt blocks recovery.
  • Recovery never writes Redis, extends TTL, populates process-local cache, or enters shadow validation.
  • Successful ordinary and clean-miss shadow fills use physical PX M when opted in; all serving and shadow reads still enforce F.
  • Cached undefined, compression envelopes, request-local ownership, coalescing on/off, ramp-down shadowing, and future-stamped frame behavior have dedicated regressions.

Observability

Add one optional bounded observer:

staleRecovery?({ outcome })

Outcomes are served, miss, read_error, read_timeout, and deserialization_error. Prometheus exposes dialcache_stale_recovery_counter; Datadog exposes dialcache.stale_recovery.count. Existing fallback errors and duration remain truthful even when retained data ultimately reaches the caller. Observer failures remain isolated.

Compatibility and rollout

This intentionally reuses the existing frame-v1 key and is not compatible with old semantic Redis clients.

Roll out readers first:

  1. deploy the age-aware library and adapters everywhere while M remains omitted or 0;
  2. verify the whole reader fleet enforces F;
  3. enable positive M for selected use cases;
  4. monitor Redis CPU, memory, evictions, source failures, and recovery outcomes.

An old reader relies on physical expiry and could serve retained F..M data as fresh. Once any writer uses PX M, do not restore an old reader until the largest enabled M has elapsed since the final such write, or the affected keys have been isolated/removed. Disabling recovery on new readers is safe because they continue enforcing F.

Cost model and benchmark

  • Steady-state read: one pipeline/batch RTT, two top-level commands (GET/MGET + TIME).
  • Steady-state write: one pipeline/batch RTT, native SET + a header-only stamp script.
  • Source rejection with recovery enabled: one additional read pair.
  • Payload bytes never cross the Lua boundary; Redis CPU remains payload-sensitive only in native value commands.

The checked-in no-threshold benchmark asserts compression, physical TTL near M, logical expiry at F, exact source/recovery counts, and coalesced fanout while reporting INFO commandstats, network deltas, and client throughput. One local Redis 8.8 / Node 22.22 run with a 64 KiB compressible JSON value observed:

Scenario Ops GET/op TIME/op Server CPU/op
Fresh end-to-end hit 200 1.00 1.00 0.28 us
Logical-stale adapter miss 200 1.00 1.00 0.55 us
End-to-end stale recovery 200 2.00 2.00 0.68 us
Coalesced stale recovery 500 callers one shared chain one shared chain observational

These loopback figures are directional, not production capacity promises; use pnpm benchmark:stale-on-error on the target engine and workload.

Validation

  • Node 22.22: corepack pnpm check
    • typecheck
    • 26 unit files / 589 tests
    • 97.62% line coverage
    • ESM/CJS build
    • packed TypeScript, ESM, and CommonJS consumer tests
  • corepack pnpm test:integration
    • 2 integration files / 153 passed
    • Redis 6.2 and Valkey 8
    • node-redis and Valkey GLIDE
    • 2 environment-dependent GLIDE Cluster cases skipped locally
  • pnpm benchmark:stale-on-error — semantic assertions passed against Redis 8.8
  • Reduced pnpm benchmark:request-local — all 10 semantic scenarios passed
  • git diff --check
  • Two independent final review rounds: clean

lan17 added a commit that referenced this pull request Aug 7, 2026
## Summary

Replace read-side Lua with native Redis commands and decode DialCache's
frame in TypeScript:

- untracked reads use `GET`
- tracked reads use one atomic, primary-routed `MGET` for the value and
watermark
- write and invalidation remain Lua-backed; a watermark-fenced tracked
write now atomically unlinks the stale value it rejects
- node-redis registers only the three mutation scripts, and GLIDE owns
only the three mutation script handles
- custom adapters can reuse the public `decodeRedisFrame` and
`decodeTrackedRedisFrame` helpers

This removes the Redis-to-Lua payload materialization and `string.sub`
copy on every hit while preserving the semantic
`DialCacheRedisClient.read()` boundary.

## Read architecture

| Adapter / mode | Untracked | Tracked | Primary guarantee |
| --- | --- | --- | --- |
| node-redis standalone | `GET` | `MGET` | standalone connection |
| node-redis Cluster | `GET` | raw `MGET` | `sendCommand(..., false,
...)` routes to the slot primary |
| GLIDE standalone | `GET` | one-command `Batch(false).mget(...)` |
standalone batches execute on the primary even with replica reads
configured; `MGET` itself is atomic |
| GLIDE Cluster | `GET` | custom-command `MGET` | explicit
`primarySlotKey` route |

The shared decoder:

- validates the frame version and minimum length
- preserves missing/short/unsupported frames as clean misses
- parses integer and fractional legacy watermarks with the same accepted
grammar as Lua
- rejects values whose Redis-created timestamp is at or before the
watermark
- preserves unsupported payload encodings as
`DialCacheRedisPayloadEncodingError`
- returns binary payloads through a zero-copy `Buffer.subarray()` view

Tracked value and watermark reads retain one atomic snapshot, with both
values returned by a single `MGET`. Their existing shared Cluster hash
tag remains required; mismatched tags still fail with `CROSSSLOT`.

## Breaking change

- `READ_CACHE_SCRIPT` and `READ_TRACKED_CACHE_SCRIPT` are removed from
`dialcache/redis-protocol`.
- `dialcacheRedisScripts.dialcacheRead` and
`dialcacheRedisScripts.dialcacheReadTracked` are removed from
`dialcache/node-redis`.
- Custom node-redis wrappers must expose native `get` / `sendCommand`;
`legacyMode` clients are unsupported because neither their callback
surface nor `.v4` view exposes the complete
native-command-plus-custom-script contract.
- The GLIDE helper requires GLIDE 2.x, a direct official `GlideClient`
or `GlideClusterClient`, and the same module namespace that created it.
Forwarding wrappers should implement `DialCacheRedisClient` directly
because their topology cannot be inferred safely.
- Official node-redis clients and direct GLIDE 2.x clients passed
through the documented helpers keep the same application-facing call
shape, so those consumers can bump the package without code changes.
- Redis keys, frame format, and invalidation behavior are unchanged. A
tracked write rejected by an active future watermark still returns
`false`, but now also unlinks the stale value key. No data migration or
cache flush is required.
- The fenced-write cleanup requires `UNLINK` (Redis 4.0+ or compatible
Valkey) and permission for scripts to invoke it. With a
command-restricted ACL that denies `UNLINK`, the write fails open as
`cache_write` and leaves the stale value for a later cleanup or expiry.

`BREAKING CHANGE:` the four deprecated read-Lua exports and
registrations above are removed; node-redis adapters require the
promise-mode native-command surface; the GLIDE helper requires a direct
GLIDE 2.x client from the supplied runtime; and the fenced-write cleanup
requires Redis `UNLINK` support plus ACL permission. Under the
repository's release configuration, this change should release as
`v1.0.0`.

## Adapter behavior changes

- The node-redis factory now requires native `get` and `sendCommand`
methods in addition to the three registered mutation methods.
- The GLIDE factory declares an optional `@valkey/valkey-glide ^2.0.0`
peer, validates `Batch` support eagerly, and classifies standalone
versus cluster behavior from the supplied runtime's client identities
before allocating scripts. Its standalone non-atomic primary batch
avoids consuming caller-owned `WATCH` state.
- Redis `MGET` returns `null` for wrong-type members. A tracked
wrong-type value is therefore a clean miss and may be repaired with a
valid DialCache frame after fallback succeeds, while a wrong-type
watermark prevents the tracked write from succeeding. An untracked `GET`
still surfaces `WRONGTYPE`. Real-engine tests cover both repair and
repeated fail-open behavior, including metrics.
- The public read contract now specifies frame decoding, miss and
watermark rules, atomic authoritative snapshots, and returned-buffer
ownership. Shared decoders validate leaf reply types; adapters retain
only client-specific envelope validation.

## Benchmark

The benchmark harness and JSON results were intentionally kept outside
the repository. Methodology:

- Redis 6.2.22 and Valkey 8.1.8
- Node 22.22.0, node-redis 4.7.1, GLIDE 2.4.2
- binary payloads of 100 B, 1 KiB, 10 KiB, 100 KiB, and 1 MiB
- fresh untracked hit, fresh tracked hit, and invalidated tracked miss
- three alternating rounds, one command in flight, loopback Docker
- median throughput, latency, Redis `INFO commandstats` execution time,
and network bytes

At 1 MiB, native fresh-hit throughput improved 15-45% across the two
engines and adapters. Server-reported command execution time per logical
read fell 95-98%. Small 100 B / 1 KiB end-to-end results were mostly
flat/noisy while reported command time still fell about 80-90%; the
notable small-case regression was Redis/node-redis's 100 B tracked hit
at about -10% throughput. These loopback, one-in-flight results are
directional rather than production-capacity measurements.

Representative Redis 6.2 + node-redis medians:

| 1 MiB scenario | Lua ops/s | Native ops/s | Lua server us/read |
Native server us/read | Lua -> native p50 |
| --- | ---: | ---: | ---: | ---: | ---: |
| untracked hit | 230 | 269 | 719.8 | 32.6 | 3.718 ms -> 2.955 ms |
| tracked hit | 217 | 259 | 713.7 | 31.2 | 3.630 ms -> 3.016 ms |
| invalidated tracked miss | 1,762 | 284 | 361.6 | 31.0 | 0.566 ms ->
2.949 ms |

The invalidated-miss row is the main tradeoff: Lua returns only a null
reply, while native `MGET` transfers the stale frame before TypeScript
rejects it. At 1 MiB this changes roughly 3-5 response bytes into about
1.05 MB. Across both engines and adapters, invalidated-miss throughput
fell 77-84% at 1 MiB (46-58% at 100 KiB), even though server-reported
command time still fell 91-94%.

The benchmark intentionally measured the read itself and therefore
includes that full transfer. In the application path, the first
completed fallback that reaches a still-fenced tracked write now
atomically unlinks the stale value, bounding subsequent transfers for
that entry. This is only a partial mitigation: a read failure or timeout
never reaches the write-side cleanup, so the stale payload can continue
to transfer or time out until another completed read cleans it up or its
TTL expires.

## Scope

This branch is updated onto the current `v0.15.0` read contract,
including the untracked-cache shadowing changes from
#122. It deliberately does not
include the server-time / maximum-age behavior proposed in
#121. That work can be evaluated
separately against this read path and its benchmark tradeoffs.

## Validation

- `corepack pnpm typecheck`
- `corepack pnpm test` - 424 tests, coverage thresholds passed
- `corepack pnpm build`
- `corepack pnpm test:package` - including real node-redis and GLIDE
standalone and Cluster consumer types, plus packed ESM/CommonJS absence
checks for all four removed APIs
- `corepack pnpm test:integration` - 113 tests across Redis 6.2, Valkey
8, and Redis Cluster
- tracked wrong-type value repair and repeated wrong-type watermark
fail-open behavior exercised end to end across both adapters and both
standalone engines
- stale tracked frames exercise the real decoder and record a remote
miss, request/get/fallback timing, and no read error across both
adapters and both standalone engines
- fenced tracked writes prove stale-value unlinking while preserving the
exact watermark and its TTL trajectory
- cluster `SCRIPT FLUSH` recovery proves mutation scripts repopulate
every master and a subsequent identical read is a cache hit
- GLIDE package tests compile against the supported 2.0.0 floor and
exercise separate module instances plus packed ESM/CommonJS error
identity
- focused GLIDE primary/replica probe and three-node Cluster probe
- `git diff --check`
@lan17
lan17 force-pushed the agent/stale-on-error branch from a2f55ce to 388027c Compare August 19, 2026 23:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add opt-in stale-on-error recovery from retained Redis values

1 participant